Add support for aggregation profiler with concurrent aggregation by ticheng-aws · Pull Request #8801 · opensearch-project/OpenSearch

ticheng-aws · 2023-07-20T17:52:47Z

Description

Add support for aggregation profiler with concurrent aggregation.

Related Issues

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-07-20T18:15:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20606/
CommitID: a89699f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-20T18:28:41Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20607/
CommitID: 8b8b7a6
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-20T18:39:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20611/
CommitID: 2f08998
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…nsearch-project#8801) Signed-off-by: Ticheng Lin <ticheng@amazon.com>

github-actions · 2023-07-20T19:43:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20617/
CommitID: 186541b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-27T21:50:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21168/
CommitID: 2dbc318
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-28T01:55:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21203/
CommitID: e6db3e2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-28T02:39:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21205/
CommitID: ab9df87
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-28T21:13:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21318/
CommitID: 6d7a314
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

reta · 2023-07-28T22:26:16Z

LGTM, thanks a lot @ticheng-aws ! But we have test failures ...

github-actions · 2023-07-29T01:51:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21335/
CommitID: 6e57e98
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…nsearch-project#8801) Signed-off-by: Ticheng Lin <ticheng@amazon.com>

…current aggregation (opensearch-project#8801) Signed-off-by: Ticheng Lin <ticheng@amazon.com>

github-actions · 2023-07-29T03:45:16Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21338/
CommitID: 5497508
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

sohami

@ticheng-aws Thanks for the refactor and adding the tests. Have few minor comments

github-actions · 2023-07-29T05:23:28Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21340/
CommitID: c29abd5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Ticheng Lin <ticheng@amazon.com>

ticheng-aws · 2023-07-29T06:01:46Z

LGTM, thanks a lot @ticheng-aws ! But we have test failures ...

Hi @reta, the failed test appears to be flaky. I have successfully passed the test on my local environment.

Tests with failures:
 - org.opensearch.search.query.QueryProfilePhaseTests.testCollapseQuerySearchResults {p0=5 p1=org.opensearch.search.query.ConcurrentQueryPhaseSearcher@2676dc05}

sohami · 2023-07-29T06:24:00Z

LGTM, thanks a lot @ticheng-aws ! But we have test failures ...

Hi @reta, the failed test appears to be flaky. I have successfully passed the test on my local environment.
Tests with failures:
 - org.opensearch.search.query.QueryProfilePhaseTests.testCollapseQuerySearchResults {p0=5 p1=org.opensearch.search.query.ConcurrentQueryPhaseSearcher@2676dc05}

@ticheng-aws You need to add this condition in the constructor of TestSearchContext

this.concurrentSegmentSearchEnabled = searcher != null && (searcher.getExecutor() != null);

Signed-off-by: Ticheng Lin <ticheng@amazon.com>

github-actions · 2023-07-29T06:31:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21341/
CommitID: 7fd4d69
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-29T07:12:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex

URL: https://build.ci.opensearch.org/job/gradle-check/21342/
CommitID: 289208e
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-07-31T17:07:13Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8801-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 35662f06bbef75447ba6204bc6a7e032066509cc
# Push it to GitHub
git push --set-upstream origin backport/backport-8801-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8801-to-2.x.

reta · 2023-07-31T17:11:13Z

@ticheng-aws needs manual backport to 2.x sadly :(

ticheng-aws · 2023-08-01T05:23:35Z

@ticheng-aws needs manual backport to 2.x sadly :(

Hi @reta, Sure, no problem. Please find the backport PRs below:

[Backport] [2.x] [Concurrent Segment Search]: Add support for aggregation profiler with concurrent aggregation (#8801) #9015
Update version guards to deal for aggregation profiler with concurren… #9017

ticheng-aws force-pushed the cs-agg-profiler branch from a89699f to 8b8b7a6 Compare July 20, 2023 17:56

ticheng-aws force-pushed the cs-agg-profiler branch from 8b8b7a6 to 2f08998 Compare July 20, 2023 18:25

ticheng-aws added a commit to ticheng-aws/OpenSearch that referenced this pull request Jul 20, 2023

Add support for aggregation profiler with concurrent aggregation (ope…

186541b

…nsearch-project#8801) Signed-off-by: Ticheng Lin <ticheng@amazon.com>

ticheng-aws force-pushed the cs-agg-profiler branch from 2f08998 to 186541b Compare July 20, 2023 19:23

sohami requested changes Jul 27, 2023

View reviewed changes

reta reviewed Jul 28, 2023

View reviewed changes

Comment thread server/src/main/java/org/opensearch/search/internal/ContextIndexSearcher.java Outdated

reta reviewed Jul 28, 2023

View reviewed changes

Comment thread server/src/main/java/org/opensearch/search/profile/Profilers.java Outdated

reta reviewed Jul 28, 2023

View reviewed changes

Comment thread ...r/src/main/java/org/opensearch/search/profile/aggregation/ConcurrentAggregationProfiler.java Outdated

ticheng-aws added 2 commits July 28, 2023 20:36

Add support for aggregation profiler with concurrent aggregation (ope…

93cc736

…nsearch-project#8801) Signed-off-by: Ticheng Lin <ticheng@amazon.com>

Address review comments for support for aggregation profiler with con…

f53ead2

…current aggregation (opensearch-project#8801) Signed-off-by: Ticheng Lin <ticheng@amazon.com>

sohami requested changes Jul 29, 2023

View reviewed changes

Comment thread server/src/main/java/org/opensearch/search/profile/ProfileResult.java Outdated

Comment thread server/src/main/java/org/opensearch/search/profile/ProfileResult.java Outdated

Comment thread .../test/java/org/opensearch/search/profile/aggregation/ConcurrentAggregationProfilerTests.java

Refactor ProfileResult class and add more tests

7fd4d69

Signed-off-by: Ticheng Lin <ticheng@amazon.com>

Fix flaky QueryProfilePhaseTests.testCollapseQuerySearchResults test

289208e

Signed-off-by: Ticheng Lin <ticheng@amazon.com>

sohami approved these changes Jul 31, 2023

View reviewed changes

reta reviewed Jul 31, 2023

View reviewed changes

Comment thread server/src/main/java/org/opensearch/search/profile/ProfileResult.java

reta approved these changes Jul 31, 2023

View reviewed changes

ticheng-aws mentioned this pull request Aug 1, 2023

[Backport] [2.x] [Concurrent Segment Search]: Add support for aggregation profiler with concurrent aggregation (#8801) #9015

Merged

6 tasks

sohami mentioned this pull request Aug 1, 2023

[DOC] Update search profile API documentation with concurrency related changes opensearch-project/documentation-website#4281

Closed

4 tasks

ticheng-aws mentioned this pull request Aug 2, 2023

[Concurrent Segment Search] Add support for profile option with concurrent aggregation #7354

Closed

Conversation

ticheng-aws commented Jul 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions Bot commented Jul 20, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jul 20, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jul 20, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jul 20, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jul 27, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 28, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jul 28, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 28, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

reta commented Jul 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 29, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jul 29, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

sohami left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 29, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

ticheng-aws commented Jul 29, 2023

Uh oh!

sohami commented Jul 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 29, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jul 29, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Uh oh!

opensearch-trigger-bot Bot commented Jul 31, 2023

Uh oh!

reta commented Jul 31, 2023

Uh oh!

ticheng-aws commented Aug 1, 2023

Uh oh!

Reviewers

Assignees

Labels

ticheng-aws commented Jul 20, 2023 •

edited

Loading

reta commented Jul 28, 2023 •

edited

Loading

sohami commented Jul 29, 2023 •

edited

Loading